Certain video communication applications allow the sharing of “content”. The word can refer to any visual content that is not the video stream of one of the participants. Examples include the contents of a computer's screen—either the entire screen (“desktop”) or a portion thereof or of a window where one of the computer's applications may be displaying its output.
Some systems used a “document camera” to capture such content. This camera would be positioned so that it would image a document placed on a table or special flatbed holder, and would capture an image of the document for distribution to all session participants. In modern systems, where computers are the primary business communication tool, the document camera is replaced with a VGA input, so that any VGA video-producing device can be connected. In advanced systems, the computer can directly interface with the video communication system so that it directly transmits the relevant content material to the session, without the need for conversion to VGA or other intermediate analog or digital format.
On one end of the spectrum, content sharing may be completely passive (“passive content sharing”). In this scenario the video communication system will only encode and transmit the content to the participants, without providing the capability to modify it in any way. When content is driven by a computer, e.g., sharing a page of a document, it may be possible to show the cursor as well as any highlighting that is applied by the underlying software. This, however, is captured as imagery—it is not possible, in other words, for a remote participant to “take over” the cursor and perform remote editing of the document. This is the mode used in many video communication applications.
On the other end of the spectrum there are distributed collaboration applications, such as shared whiteboards, and sometimes referred to as “active content sharing.” In this scenario, users are able to collaboratively edit and view a document in a synchronized fashion. The complexity in building such systems is significant, and requires specialized protocols and applications. Oftentimes, users are not able to use their favorite applications and are forced to use special, network-aware, programs (typically of lower sophistication). Thus, video communication applications can be using passive content sharing rather than active.
Certain video communication systems that rely on the Multipoint Control Unit (MCU) architecture, such as those using the ITU-T Rec. H.323 standard, “Packet-based multimedia communications systems,” incorporated herein by reference in its entirety, also can support a single content stream. ITU-T Rec. H.239, “Role management and additional media channels for H.3xx-series terminals”, incorporated herein by reference in its entirety, defines mechanisms through which two video channels can be supported in a single H.323 session or call. The first channel is used to carry the video of the participants, and the second to carry a PC graphics presentation or video. For presentations in multipoint conferencing, H.239 defines token procedures to guarantee that only one endpoint in the conference sends the additional video channel which is then distributed to all conference participants.
When an H.323 call is connected, signaling defined in ITU-T Rec. H.245 can be used to establish the set of capabilities for all connected endpoints and MCUs. When the set of capabilities includes an indication that H.239 presentations are supported, a connected endpoint can choose to open an additional video channel. First the endpoint has to request a token from the MCU. The MCU can then check if there is another endpoint currently sending an additional video channel. The MCU will use token messages to make this endpoint stop sending the additional video channel. Then the MCU will acknowledge the token request from the first endpoint which then can begin to send the additional video channel which, as an example, may contain encoded video from a computer's video output at XGA resolution. Similar procedures can be defined for the case when two endpoints are directly connected to each other without an intermediate MCU.
Certain video communication systems used for traditional videoconferencing involve a single camera and a single display for each of the endpoints. High-end systems for use in dedicated conferencing rooms, may feature multiple monitors. The 2nd monitor is often dedicated to content sharing. When no such content is used, one monitor may feature the loudest speaker whereas the other monitor shows some or all of the remaining participants. When only one monitor is available, then either content has to be switched between video, or the screen must be split between the two.
Video communication systems that run on personal computers (or tablets or other general-purpose computing devices) typically have more flexibility in terms of how they display both video and content, and can also become sources of content sharing. Indeed, any portion of the computer's screen can be indicated as source for content and be encoded for transmission without any knowledge of the underlying software application (“screen dumping”, as allowed by the display device driver and operating system software). Inherent system architecture limitations, such as with H.300-series specifications, where only two streams (one video and one content) are allowed, may prohibit otherwise viable operating scenarios (multiple video streams and multiple content streams).
So-called “telepresence” can convey the sense of “being in the same room” as the remote participant(s). In order to accomplish this goal, these systems can utilize multiple cameras as well as multiple displays. The displays and cameras are positioned at carefully calculated positions in order to be able to give a sense of eye-contact. Some systems involve three displays—left, center, and right—although configurations with only two or more than three displays are also available.
The displays can be situated in carefully selected positions in the conferencing room. Looking at each of the displays from any physical position on the conferencing room table is supposed to give the illusion that the remote participant is physically located in the room. This can be accomplished by matching the exact size of the person as displayed to the expected physical size that the subject would have if he or she were actually present in the perceived position within the room. Some systems go as far as matching the furniture, room colors, and lighting, to further enhance the life-like experience.
In order to be effective, telepresence systems should offer very high resolution and operate with very low latency. For example, these systems can operate at high definition (HD) 1080p/30 resolutions, i.e., 1080 horizontal lines progressive at 30 frames per second. To eliminate latency and packet loss, they also use dedicated multi-megabit networks and typically operate in point-to-point or switched configurations (i.e., they avoid transcoding).
Some video conferencing systems assume that each endpoint is equipped with a single camera, although they can be equipped with several displays.
For example, in a two-monitor system, the active speaker can be displayed in the primary monitor, with the other participants shown in the second monitor in a matrix of smaller windows. One matrix layout, referred to as “continuous presence”, permits participants to be continuously present on the screen rather than being switched in and out depending on who is the active speaker. In sessions with a large number of participants, when the size of the matrix is exhausted (e.g., 9 windows for a 3×3 matrix) then participants can be entered and removed from the continuous presence matrix based on least-recently active audio policy. The layout is still referred to as “continuous presence” in this case as well.
A similar configuration to the continuous presence layout is the preferred speaker layout, where one (or a small set of speakers) is designated as the preferred one and is shown in a larger window than the other participants (e.g., double the size).
An alternative way is to use the second monitor to display content (e.g., a slide presentation from a computer) and the primary monitor to show the participants. The primary monitor then is treated as with a single-monitor system. The primary monitor can feature a preferred speaker layout as well. In this case, the preferred speaker is shown in larger size in the primary monitor, together with a number of other participants in smaller sizes, whereas content is shown in the second monitor.
Telepresence systems that feature multiple cameras can be designed so that each camera is assigned to its own codec. A system with three cameras and three screens would then use three separate codecs to perform encoding and decoding at each endpoint. These codecs would make connections to three counterpart codecs on the remote site, using proprietary signaling or proprietary signaling extensions to existing protocols.
The three codecs are typically identified as “left,” “right,” and “center.” In this document such positional references are made from the perspective of a user of the system; left, in this context, is the left-hand side of a user that is sitting in front of the camera(s) and is using the system. Audio, e.g., stereo, and can be handled through the center codec. In addition to the three video screens, telepresence systems can include a fourth screen to display computer-related content such as presentations. This can be referred to as the “content” or “data” stream.
FIG. 1 depicts the architecture of a commercially available legacy telepresence system (the Polycom TPX 306M). The system features three screens (plasma or rear screen projection) and three HD cameras. Each HD camera is paired with a codec which is provided by an HDX traditional (single-stream) videoconferencing system. One of the codecs is labeled as Primary. Notice the diagonal pairing of the HD cameras with the codecs. This is so that the correct viewpoint is offered to the viewer on the remote site.
The Primary codec is responsible for audio handling. The system here is shown as having multiple microphones, which are mixed into a single signal that is encoded by the primary codec. There is also a fourth screen to display content. The entire system is managed by a special device labeled as the Controller. In order to establish a connection with a remote site, this system performs three separate H.323 calls, one for each codec. This is because existing ITU-T standards do not allow the establishment of multi-camera calls. This architecture is typical of certain telepresence products that use standards-based signaling for session establishment and control. Use of the TIP protocol would allow system operation with a single connection, and would make possible up to 4 video streams and 4 audio streams to be carried over two RTP sessions (one for audio and one for video).
Referring to FIG. 1 content is handled by the Primary codec (notice that the Content display is connected to the Primary codec). The Primary codec will use H.239 signaling to manage the content display. A legacy, non-telepresence, two-monitor system is configured essentially in the same way as the Primary codec of a telepresence system.
Telepresence systems pose unique challenges compared with traditional videoconferencing systems. One challenge is that such systems be able to handle multiple video streams. A typical videoconferencing system only handles a single video stream, and optionally an additional “data” stream for content. Even when multiple participants are present, the Multipoint Control Unit (MCU) is responsible for compositing the multiple participants in a single frame and transmitting the encoded frame to the receiving endpoint. Certain systems address this in different ways. One way is to establish as many connections as there are video cameras, e.g., for a three camera systems, three separate connections are established, and mechanisms are provided to properly treat these separate streams as a unit, i.e., as coming from the same location.
A second way is to use extensions to existing signaling protocols, or use new protocols, such as the Telepresence Interoperability Protocol (TIP). TIP is currently managed by the International Multimedia Telecommunications Consortium (IMTC); the specification can be obtained from IMTC at the address 2400 Camino Ramon, Suite 375, San Ramon, Calif. 94583, USA or from the web site http://www.imtc.org/tip. TIP allows multiple audio and video streams to be transported over a single RTP (Real-Time Protocol, RFC 3550) connection. TIP enables the multiplexing of up to four video or audio streams in the same RTP session, using proprietary RTCP (Real-Time Control Protocol, defined in RFC 3550 as part of RTP) messages. The four video streams can be used for up to three video streams and one content stream.
In both traditional as well as telepresence system configurations, content handling is thus simplistic. There are inherent limitations of the MCU architecture, in both its switching and transcoding configurations. The transcoding configuration introduces delay due to cascaded decoding and encoding, in addition to quality loss, and is thus problematic for a high-quality experience. Switching, on the other hand, can become awkward, such as when used between systems with a different number of screens.
Scalable video coding (‘SVC’), an extension of the well-known video coding standard H.264 that is used in certain digital video applications, is a video coding technique that has proven to be effective in interactive video communication. The bitstream syntax and decoding process are formally specified in ITU-T Recommendation H.264, and particularly Annex G. ITU-T Rec. H.264, incorporated herein by reference in its entirety, can be obtained from the International telecommunications Union, Place de Nations, 1120 Geneva, Switzerland, or from the web site www.itu.int. The packetization of SVC for transport over RTP is defined in RFC 6190, “RTP payload format for Scalable Video Coding,” incorporated herein by reference in its entirety, which is available from the Internet Engineering Task Force (IETF) at the web site http://www.ietf.org.
Scalable video and audio coding has been used in video and audio communication using the so-called Scalable Video Coding Server (SVCS) architecture. The SVCS is a type of video and audio communication server and is described in commonly assigned U.S. Pat. No. 7,593,032, “System and Method for a Conference Server Architecture for Low Delay and Distributed Conferencing Applications”, as well as commonly assigned International Patent Application No. PCT/US06/62569, “System and Method for Videoconferencing using Scalable Video Coding and Compositing Scalable Video Servers,” both incorporated herein by reference in their entirety. It provides an architecture that allows for high quality video communication with high robustness and low delay.
Commonly assigned International Patent Application Nos. PCT/US06/061815, “Systems and methods for error resilience and random access in video communication systems,” PCT/US07/63335, “System and method for providing error resilience, random access, and rate control in scalable video communications,” and PCT/US08/50640, “Improved systems and methods for error resilience in video communication systems,” all incorporated herein by reference in their entirety, further describe mechanisms through which a number of features such as error resilience and rate control are provided through the use of the SVCS architecture.
In one example, the SVCS operation includes receiving scalable video from a transmitting endpoint and selectively forwarding layers of that video to the receiving participant(s). In a multipoint configuration, and contrary to an MCU, this exemplary SVCS performs no decoding/composition/re-encoding. Instead, all appropriate layers from all video streams can be sent to each receiving endpoint by the SVCS, and each receiving endpoint is itself responsible for performing the composition for final display. Note that this means that, in the SVCS system architecture, all endpoints can have multiple stream support, because the video from each transmitting endpoint is transmitted as a separate stream to the receiving endpoint(s). Of course, the different streams can be transmitted over the same RTP session (i.e., multiplexed), but the endpoint should be configured to receive multiple video streams, decode, and compose them for display. This is an important advantage for SVC/SVCS-based systems in terms of the flexibility of handling multiple streams.
In systems that use the SVC/SVCS architecture, content sharing can work as follows. The user interface of the endpoint application which runs on a personal computer can allow the user to select any existing application window for sharing with the other participants. When such a window is selected, it can appear in the list of available “shares” in the user interface of the other users. To alert them to the new share if no share is currently shown in their window, the newly introduced share can be shown in a “preferred view” (i.e., larger size view) in the main application window together with the videos of the session participants (i.e., the same way as a video participant). Since the size of this view may be small, and at any rate smaller than the size of the typical application window, the user can double-click on it so that it “pops-out” into its own window and thus allow the user to freely resize it. In a room-based system with two monitors the content can be shown in its own monitor; if only one monitor is available then the screen can be split between video windows and the content window.
When the shared content is viewed by one or more of the participants, the originating endpoint can encode and transmit the content in the same way that it does any other source of video. Of course, the video encoding and decoding may be different in order to accommodate the particular features of computer-generated imagery, but from a system perspective the content stream is treated as any other video stream. Note that the same video encoder can be used for content as well, but with different tuning and optimization settings (e.g., lower frame rate, higher spatial resolution with finer quantization, etc.). The system can support multiple content shares per endpoint. Although it may be confusing for the end-user to have multiple active content shares, the system architecture can support it. The inherent multi-stream support of the SVCS architecture makes content handling a natural extension of video.
Commonly assigned International Patent Application No. PCT/US11/038003, “Systems and Methods for Scalable Video Communication using Multiple Cameras and Multiple Monitors,” incorporated herein by reference in its entirety, describes systems and methods for video communication using scalable video coding with multiple cameras and multiple monitors. In this case the architecture is expanded to include multiple video displays and possibly multiple sources for a particulate endpoint.
It can be desirable to improve the way content is handled, so that users can have improved interaction with the content without adversely increasing the complexity of the implementation or radically changing the underlying system architecture.